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ABSTRACT 

A study investigated the reading skills of entering 
students at the Open University. Just over 2,000 of the 3,000 
students (representing 107. of all students taking courses at the 
foundation level) completed questionnaires that were mailed to them. 
Subjects completed 3 cloze tests, graded in difficulty, and a 
vocabulary test. Results indicated that (1) most students showed at 
least partial comprehension on the 2 simpler cloze tests; (2) 95/. of 
students showed some degree of inadequacy in their comprehension of 
the cloze passage that represented fairly typical academic prose; (3) 
to be comfortable with prose in the most difficult cloze test, 
students would need to score about 707. or more on the vocabulary 
test — most scored below the 70% level; and (4) arts students had the 
highest scores, and technology students the lowest (differences were 
slight but statistically significant). Future research will follow 
the students through their careers at the Open University. (Contains 
21 references, and 8 tables and 12 figures of data.) (RS) 
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Abstract 

As explained in our previous report (Scott and Macdonald-Ross 1995), we used a postal 
survey to test the reading skills of a sample of students entering the Open University at 
foundation level. The sample was structured to include approximately equal numbers 
from the five foundation courses (Arts, Social Sciences, Mathematics, Science, Technology) 
and from four levels of ‘educational qualifications on entry’. 

The test materials were sent out to 3000 students; just over 2000 completed forms were 
returned. The data has now' been collated and analysed. Results suggest that many 
entering students may have some difficulty comprehending academic texts. The results 
also show significant differences between students grouped by faculty and by level of 
educational qualification. 

The progress of the cohort of students sampled is being monitored throughout the 
foundation year. 



Introduction 

During the first 25 years of the Open University’s existence, no assessment of entering 
students’ basic skills was carried out. In fact, almost no psychometric data exists for any 
of the OU’s student population. One exception is a study of younger OU students (W oodley 
and McIntosh 1980) and we refer to some of that data later in this report. Social and 
personal data about students and courses is gathered routinely by theOU application form 
and by IET’s Student Research Centre. 

Each year many students withdraw from or fail their courses (though in general few 
students fail if they survive to take the exam). Surveys of students’ reported reasons for 
withdrawal have been carried out (Woodley and Parlett 1983, Woodley 1 294). Rarely do 
students say they found courses too difficult, or the prose too difficult t > read. Most 
frequently, they cite workload or domestic problems. Without objective data, which the 
present work starts to provide for the first time, any assessment of the importance of good 
basic skills levels (as a factor in student success and course retention rates) could only be 
speculative. 

As a first step in gathering relevant data, we surveyed the reading skills of a large sample 
of students entering the OU at foundation level and are monitoring their progress through 
the first year of their studies with the OU. 

The tests we used have been described in detail in Scott and Macdonald-Ross 1 995. In this 
report, we describe the structure of the sample and analyse the data so far gathered and 
discuss the implications. 

Michael Macdonald-Ross. Reader in Textual Communication. 

Bernard Scott. Research Fellow. 
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The Reading Tests 

The tests consisted of three ‘cloze’ passages and a vocabulary test (details in Scott and 
Macdonald-Ross 1995). In a typical cloze task, every fifth or sixth word is deleted from a 
passage and the subject has to insert what he or she considers to be the most appropriate 
word Cloze is particularly suitable for a postal survey since it is relatively robust with respect 
to time allowed for completion, as long is this is above a sensible minimum This was one 
good reason to allow students to complete the tests in their own time; we were also influenced 
by our knowledge of the general probity and reliability of OU students. 

In cloze tests of this kind, scores of about 60% or above show the respondent can read and 
work on the passage satisfactorily, without assistance. Scores between 40 /o and 60 /o show 
partial comprehension, or the ability to read and work on the passage under supervision. 
Scores below about 40% show inadequate comprehension (these levels based mainly on 
work discussed in Bormuth 1967, 1968). The tnree passages were chosen to be at three 
different levels of difficulty. Figure 9 shows the differences between the tests as shown by the 
Flesch Reading Ease (RES) scale, a commonly used measure of text readability . RES scores 
are known to be good predictors of text comprehensibility (see section on readability below). 
Passage 1 is part of a basic skills screening test (ALBSU 1993), designed to identify readers 
who have trouble with basic functional literacy. Passage 2, also from an ALBSU source, is 
a more difficult passage, roughly at the same level of difficulty as editorials in middlebrow 
popular newspapers. Passage 3 was chosen as an example of fairly typical academic text . It 
is an extract from a book review in the Times Higher Education Supplement (Turner 1 994). 
The vocabulary test was included as a useful, quick-to-do additional task. Vocabulary 
scores are known to correlate highly with more general psychometric measures of verba 
abilities. The Woodley and McIntosh study (1980) used a vocabulary scale and showed it 
to be a good predictor of student success. 

Readability: some general points 

Readability measures of the type we use are indices that predict; they are not meant to 
explain how students learn, or to teach authors how to write. What they do and do well 
- is to predict the difficulty of prose, especially if there is some information about the reading 
skills of the readers. There are readability formulae designed for every type of prose: all ages, 
all levels of difficulty, practically every major world language, and so on There are lew 
metrics in the whole of the social sciences which have been as extensively validated as 
readability measures such as the Dale-Chall and Flesch RES; their effectiveness and technical 
reliability have been well demonstrated. 

Most formulae enshrine a balance between ease of application and coverage of linguistic 
variables. Thus, good predictions are got by 2-variable formulae where one variable is an 
index of semantic complexity (usually a direct or indirect measure of the vocaoidary 
demands of the text), and the other variable is an index of syntactic complex, y (usually 
sentence length). These two indices combined are highly predictive of prose difficulty as 
experienced by readers. “The good news is that two language variables, and simple ones at 
that account for so much of the variance in theprediction of readable writing (Klare 1984). 
More complex formula; developed for special purposes such as linguistic research have their 
uses, but add little to the accuracy of prediction. 

Because of the extent of the literature on readability, it is not a simple matter to give an 
overview of the field. There are certainly hundreds of studies on the effect of readabiht} on 
various age °roups, and of its effect under various conditions of motivation, prior knowledge 
of subject-matter, and its relation to the reading skills of subjects. The main resources to gain 
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access to this literature are the reviews by George Klare ( 1 963, 1 974 and 1 984). The topic was 
also addressed in a review of research on language by Macdonald-Ross (1979). 

A few examples of the research on readability may be helpful: Klare and Smart (1973) found 
a close relationship between readability level of correspondence course material and the 
probability that correspondence students would persevere and complete their courses. 
Murphy (1947a. b) and Swanson (1948) found that readership of newspapers goes up with 
improved readability in split-run studies of newspapers. Sticht and colleagues showed that 
readability had a significant effect on the use of military manuals (Kern, Sticht & Fox 1970), 
and Johnson et al (1972) showed the effect of readability on discrepancies in following 
military procedures. These results suggest that readability measures are as applicable to 
adults as to younger age groups. Also, the two best-known formulae, the Flesch Reading 
Ease Score and the Dale-Chall formula, were actually developed for adult materials and 
validated against standard criteria for adult readers. 

Work on readability in the OU during the 1970s 

A certain amount of work was done in the OU in the 1970s, but there were two obstacles to 
progress. The first was that all readability measures had to be applied by hand, which is 
obviously laborious, especially considering how much prose the OU produces each year. 
Nevertheless, in IET’s Textual Communication Research Group. Eleanor Smith scored all 
the four original foundation courses. This work revealed large variations in the level of 
difficulty that students had to contend with. Now, readability formulae are available on (for 
example) the Microsoft Word application, and the textual components of the more recent 
courses are available on the same software. Since automated formulae do not give quite the 
same results as the original hand formulae, we have done some work to estimate the effect 
of this discrepancy. 

Our hypothesis 

We expect that the poorer readers will preferentially drop out or fail courses. This will be 
most likely to occur in courses which are the least readable, and most heavily overloaded, 
sc that students who are weak readers will have difficulty in completing their study of course 
material within the time available to them. 

The extent of the gap between reading skills of students and the readability of course 
materials will be identified . The expectation is that large mismatches between reader and text 
will have significant consequences for the student. Lastly, we can identify the range of 
readability scores for different authors and subjects, allowing editorial and other skills to be 
brought to bear on the most difficult passages. 

The practical potential for the results of this work to be used for the benefit of the University 
and its students is considerable, especially at a time when we need to find ways of 
demonstrating the quality of our teaching systems. 

Structure of the sample 

Our reading tests were sent out to 3000 entering students. They represent approximately 
1 0% of all students taking courses at foundation level. The sample was structured to ensure 
representative numbers came from each of the five foundation courses (Arts. Social 
Sciences. Mathematics. Science. Technology) and from each of four levels of prior educational 
qualifications: low (less than O Level), hwish (O Level or single A level), medium (A Level 
in two or more subjects) and high (higher education). 

See Table 1. page 12. 



Procedure 

The full text of the survey instrument is shown in the Appendix of our earlier report (Scott 
and Macdonald-Ross 1995). As well as the reading tests proper, there is a letter of 
introduction, instructions on how to do cloze and a short cloze passage for practice, with 
correct answers supplied. 

A sample of 3000 students, structured as above, was drawn from the OU’s student database 
and the reading tests were sent out following the standard procedures of the Institute’s 
Student Survey Office. An initial mailing at the end of September 1 994 was followed up four 
weeks later by a reminder postcard to non-responders. Four weeks after that, all outstanding 
non-responders were sent a duplicate copy of the full survey instrument. 

As completed tests arrived back they were date-stamped, then processed for scoring and 
analysis. The cloze passages were hand marked against a list of correct answers. Only words 
identical to deleted items were scored as correct. This simple scoring procedure has been 
shown to be as effective as more sophisticated procedures where synonyms are accepted as 
correct (Bormuth 1975). 

The vocabulary tests were all individually checked for procedural errors and written 
responses assessed for correctness. The test presents students with a list of forty words 
ordered by frequency of occurrence. Students were asked to work through the list of words 
in the order presented and tick those they felt they understood. For the last five words they 
had ticked, they were asked to supply some form of definition to show they understood its 
meaning. If a required written definition was missing or incorrect, the count of words ticked 
was reduced by one mark. 

All test scores were entered into a computer data base for analysis. 

Results 

Results for all students on the four tests used are shown as histograms (Figures 1 -4). Means, 
standard deviations and inter-test correlations are shown in Table 2, pi 3. Interpretations 
are as follows. 

Passage 1 (Figure 1 ): as expected for a test of basic functional literacy, almost all the students 
performed well on this test. 

Passage 2 (Figure 2): almost a third of students showed partial comprehension and just over 
two thirds showed satisfactory comprehension. 

Passage 3 (Figure 3): almost a third of students showed inadequate comprehension and 
nearly two thirds showed partial comprehension. In all, about 95% of all respondents had 
some difficulty in comprehending this passage. 

Vocabulary (Figure 4): to be comfortable with prose as difficult as Passage 3 students would 
need to score at least 70% or more on the vocabulary test. As can be seen in Figure 4, about 
two-thirds of students scored below this level. 

Comparisons between faculties for Passage 3 and Vocabulary are shown graphically in 
Figures 5-6. As can be seen. Arts students, as a group, consistently performed better than 
other faculty groups, with Technology students generally performing least well. Analysis of 
variance (ANOVA) shows the differences between faculties to be significant, and highly 
significant for Passage 3 and Vocabulary (Table 3, p 1 3). Pairwise comparisons of means 
were carried out using Tukey’s Studentized Range Test, a procedure that carries out t-test 
type comparisons but with a weighting (based on the number of comparisons being made) 
that reduces the probability of finding significant differences by chance (Table 4, pi 4). 
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Passage 1: All Students 



Figure 1 



Figure 2 



Figure 3 




Passage 2: All Students 




Passage 3: All Students 




Scores 



Vocabulary: All Students 



Figure 4 




Figure 5 
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Discrimination power of the tests 

Passage 3 and the vocabulary tests discriminated between students because they were 
difficult enough to be testing without being overwhelming. Also, they fairly represent the 
standards of literacy which are expected of university-level education. All four tests point 
in the same direction, as shown by the correlations listed in Table 2, pi 3. 

Comparisons between educational levels 

Comparisons between students grouped by level of educational qualification are shown in 
figures 7 and 8. There is a consistent pattern: the higher a students’ prior qualifications, the 
higher his or her scores are likely to be. The relationship is highly significant statistically, as 
results of ANOVA show in Table 3, pi 3. Pairwise comparisons of means are shown in Table 
5, pl5. Means and standard deviations for all sub-samples are shown in Table 6, pi 6. 

Passage 3 by educ. level 



Figure 7 



Figure 8 
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Difficulty of the passages 

Opposite is a scale showing the relative 
difficulty of the three passages as 
measured by the Flesch Reading Ease 
formula, a good predictor of the difficulty 
of prose for adult readers. We will be 
measuring the readability of samples from 
the foundation course texts to see what 
level of difficulty they may pose to our 
students. 



Figure 9 In the Flesch scale the lower the 
score the more difficult the prose 



Those who did not return the survey form 

Of course, we have no scores for those who did not return the tests. However, there are indications 
that the reading skills of those who did not return the form are poorer on average than those who 
did return the form. 

One clue is provided by the comparison between early and late returns. Taking the early vs late 
returns for passage 3 (Figure 10) it is clear that the later returns are scoring the less highly. This 
difference is highly significant statistically. 

The inference is that if scores for all 3.000 were available, the average scores would be lower, 
especially for passage 3. Table 7, pl7, gives the results of ANOVA [analysis of variance] for all three 
passages and the vocabulary test related to date of return; Table 8, pi 8, gives Tukey's Studentized 
Range Test results for dates of return divided into ten groups. 

Another, different, approach is to check with final registration data. Final registration is quite 
important because it commits students to paying off the balance of their fees. The cut-off date for 
this in 1995 was April 28th. By then students had received a substantial amount of course material 
and had been required to submit two TMAs and two CMAs. 

There is a strong relationship between test scores and final registration [data to be presented in next 
report]. And. looking at Figure 12. it is clear that a much higher proportion of respondents finally 
register compared to non-respondents. Thus there is a strong inference that non-respondents would 
have scored less well than respondents, had we been able to administer the test. 

Of course, this is an inference - or an hypothesis if you wish - but not actual data. The actual data 
shows that those who do not finally register are preferentially those of lower educational rank, and 
(as we have discussed above, and shown in Figures 7 & 8) educational level correlates witli the 
reading test scores. 

Our conclusion is that the tests present a slightly rosy picture, and tend to over-estimate the skills 
of the population of entering students. Remembering the poor scores for passage 3. we can say that 
many provisionally registered students w ill have problems with the level of prose presented in the 
foundation courses. In our judgement, such difficulties are bound to be at least partially responsible 
for many drop-outs. 
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Passage 3: early & Me returns 



Figure 10 




Figure 11 
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Summary and discussion 

1 . At the basic level assessed by Passage 1, all our respondents are literate. 

2. However, most (95%) of our respondents have some limitations in the extent to which they 
can read academic prose as indexed by our Passage 3. The lowest quartile of these students 
are bound to experience prolonged difficulty in reading our course texts at the required 
speed and with the necessary comprehension to perform well on the course. To put it 
another way, most entering students enrolled for OU foundation courses have reading 
skills which are not at the appropriate level for working with academic prose, and many 
are seriously deficient. 

3. Lest it be thought that our data over-estimates the problem, we have shown some good 
reasons for thinking that the reading skills of the non-respondents may be even more limited. 

4. There are significant differences when the data are analysed by faculty. 

5. A significant relationship emerges when the data are analysed by educational level. The 
lower the educational level, the lower the mean test scores. This is important in view of the 
University’s efforts to make itself more open than hitherto. Policy makers will realise that 
their worthy intention runs against the tenor of our data, and risks an increasing 
proportion of drop-outs, with all the hidden frustration and visible waste of resources that 
this entails. There are various actions that might be proposed to reduce this tension. 

6. Having established that many students enrolled as of October 1994 are at risk of failing 
or dropping out, our task now is to monitor the progress of those who remain from our 
sample. Courses began their teaching in February 1995 and will finish, with exams, in 
October 1995. Steps along the way include the submission of tutor-marked and computer- 
marked assignments (TMAs and CMAs), final registration and attendance at summer 
school. 

In our next report we will present data from final registration figures to show that students 
scoring poorly in our tests are more likely to drop out than stuc „uts scoring well. 

7. Our other major task is to assess the readability and workload levels of a range of OU 
course This is a tractable problem once course materials are in a suitable electronic 
format to allow us to use automated procedures. 

8 . A range of further research studies are being considered and proposals are being prepared. 
These include surveys of other students on other courses. As well as surveys of reading 
skills, readability and workload, there is also a need for more fine-grained studies of how 
OU students interact with OU text-based course materials in print and on screen. The latter 
is increasingly important as resource-based learning using CD-ROMs is introduced into 
new courses. 
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Table 1 Structure of Sample: Faculties*Education Levels 

Sample size=3000; 150 subjects per cell. 

Total population at foundation level= c.30,000. 

Foundation Courses 



Education levels 
Low 
Lowish 
Medium 
High 


AD M 


S T 


Faculties: 


Arts (A) 


cl 0,000 




Social Sciences (D) 


c9,000 




Maths (M) 


c3,000 




Sciences (S) 


c4,500 




Technology (T) 


c4,500 


Education Levels: 


Low 


10% 




Lowish 


20% 




Medium 


30% 




High 


30%. 



Definitions of the four education levels: 

low = no formal qualifications; CSE (other than grade 1). RSA or school leaving certificate. 

lowish = CSE (grade 1), GCE Olevel, SCE O grade, GCSE, BEC general certificate or 
diploma;professional qualification below A level equivalent; GCE A level, SCE H grade, 
higher school certificate or equivalent in one subject. 

medium = GCE A level, SCE H grade, higher school certificate or equivalent in more than one 
subject; ONC/OND, BEC, BTEC, SCOTBEC, SCOTVEC national certificates and diplomas, 
TEC and SCOTBEC, SCOTEC higher certificates and diplomas, professional qualification 
equivalent to A level but below degree level. 

high = HNC/HND, BEC, BTEC, SCOTBEC and SCOTVEC higher national certificates and 
diplomas, TEC and SCOTEC higher certificates and diplomas, teachers certificates or 
equivalent, university diploma or equivalent, first degree, postgraduate degree, 
professional qualification equivalent to or above degree level. 

These terms are now standard usage throughout OU statistics and survey research. 
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Table 2 

Means and Standard Deviations 



All students 


total marks 


Mean 


% 


Standard Deviation 


Passage 1 


36 


29.06 


80.7 


3.15 


Passage 2 


38 


24.77 


65.2 


3.55 


Passage 3 


50 


20.96 


41.9 


6.66 


Vocabulary 


40 


23.91 


59.8 


5.59 



Mean: the arithmetic mean 

% = mean as percent of total marks available on each test. 



Correlations Between Tests 

All correlations are highly significant. 



Passage 1 and Passage 2 0.43 

Passage 1 and Passage 3 0.30 

Passage 1 and Vocabulary 0.24 

Passage 2 and Passage 3 0.34 

Passage 2 and Vocabulary 0.21 

Passage 3 and Vocabulary 0.36 



Table 3 Summary ofANOVA Results 

Passage 1 



Source of Variance 


DF 


F Value 


Significance level 


Faculty 


4 


4.62 


p<0.001 


Ed. Level 


3 


11.62 


p<0.0001 


Faculty*Ed. Level 


12 


0.62 


ns 


Passage 2 

Source of Variance 


DF 


F Value 


Significance level 


Faculty 


4 


2.42 


p<0.05 


Ed. Level 


3 


17.39 


p<0.0001 


Faculty*Ed. Level 


12 


1.33 


ns 


Passage 3 


Source of Variance 


DF 


F Value 


Significance level 


Faculty 


4 


10.53 


p<0.0001 


Ed. Level 


3 


53.88 


p<0.0001 


Faculty* Ed. Level 


12 


1.37 


ns 


Vocabulary 


Source of Variance 


DF 


F Value 


Significance level 


Faculty 


4 


20.03 


p<0.0001 


Ed. Level 


3 


26.02 


p<0.0001 


Facultv*Ed. Level 


12 


0.95 


ns 



DF = degrees of freedom 
ns = not significant 



Table 4 Comparisons of Means For Faculties (Tukey Method) 

Differences significant at p<0.05 are indicated by ***; ns = not significant. 
A positive difference indicates first mean is greater than second and vice versa. 

Passage 1 Difference Between Means 



Arts v Science 


0.19 


ns 


Arts v Social Science 


0.53 


ns 


Arts v Maths 


0.61 


ns 


Arts v Technology 


0.81 


* * * 


Science v Social Science 


0.35 


ns 


Science v Maths 


0.42 


ns 


Science v Technology 


0.62 


* * * 


Social Science v Maths 


0.08 


ns 


Social Science v Technology 


0.27 


ns 


Maths v Technology 


0.19 


ns 


Passage 2 






Arts v Science 


0.04 


ns 


Arts v Social Science 


0.28 


ns 


Arts v Maths 


0.48 


ns 


Arts v Technology 


0.60 


ns 


Science v Social Science 


0.44 


ns 


Science v Maths 


0.24 


ns 


Science v Technology 


0.56 


ns 


Social Science v Maths 


-0.19 


ns 


Social Science v Technology 


0.13 


ns 


Maths v Technology 


0.32 


ns 


Passage 3 






Arts v Science 


0.91 


ns 


Arts v Social Science 


1.12 


ns 


Arts v Maths 


1.46 


* * * 


Arts v Technology 


2.74 


* * * 


Science v Social Science 


0.21 


ns 


Science v Maths 


0.55 


ns 


Science v Technology 


1.83 


*** 


Social Science v Maths 


0.34 


ns 


Social Science v Technology 


1.62 


* * * 


Maths v Technology 


1.28 


* * * 


Vocabulary 






Arts v Science 


1.67 


sjc ^|e j)c 


Arts v Social Science 


2.23 


* * * 


Arts v Maths 


2.79 


* * * 


Arts v Technology 


2.89 


* * * 


Science v Social Science 


0.57 


ns 


Science v Maths 


1.13 


* * * 


Science v Technology 


1 .22 


* * * 


Social Science v Maths 


0.56 


ns 


Social Science v Technology 


0.66 


ns 


Maths v Technology 


0.10 


ns 




14 



lb 




Table 5 

Comparisons of Means for Educational Levels (Tukey Method) 



Comparisons significant at p<0.05 are indicated by *** 
ns = not significant. A positive difference indicates first mean 
is greater than second and vice versa. 



Difference Between Means 



Passage 1 



High v Medium 


-0.33 


ns 


High v Lowish 


0.17 


ns 


High v Low 


1.00 


** * 


Medium v Lowish 


0.21 


ns 


Medium v Low 


1.04 


** * 


Lowish v Low 


0.83 


* ** 


Passage 2 


High v Medium 


-0.15 


ns 


High v Lowish 


0.35 


ns 


High v Low 


1.33 


* ** 


Medium v Lowish 


0.50 


ns 


Medium v Low 


1.47 


* ** 


Lowish v Low 


0.97 


*** 


Passage 3 


High v Medium 


1.45 


* * * 


High v Lowish 


3.16 


* ** 


High v Low 


4.83 


** * 


Medium v Lowish 


1.71 


* * * 


Medium v Low 


3.38 


** * 


Lowish v Low 


1.67 


*** 


Vocabulary 


High v Medium 


1.58 


* * * 


High v Lowish 


1.90 


*** 


High v Low 


2.98 


** * 


Medium v Lowish 


0.32 


ns 


Medium v Low 


1.40 


*** 


Lowish v Low 


1.08 


*** 
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Table 6 

Means and Standard Deviations (Faculty*Educational Level) 



Passage 1 




Educational Level 




Faculty 


Low 


Lowish 


Medium 


High 


Arts 


29.0/2.27 


29.6/2.32 


29.7/2.19 


29.6/3.75 


Social Sciences 


28.2/3.27 


29.0/3.13 


29.2/4.19 


29.3/3.00 


Maths 


28.1 / 4.6° 


29 . 0 / 3.21 


29.4/2.87 


29.0/3.28 


Science 


28.4/2.91 


29.3/2.09 


29.6/2.56 


29.8/2.45 


Technology 


27.9/4.10 


28.9/2.99 


28.90/3.47 


29.0/2.89 


Passage 2 




Educational Level 




Faculty 


Low 


Lowish 


Medium 


High 


Arts 


23.6/4.16 


25.0/2.87 


25.6/3.13 


25.8/2.74 


Social Sciences 


23.8/3.44 


24.5/3.46 


25.2/3.67 


24.8/2.88 


Maths 


23.5/4.19 


25.0 / 3.92 


25.5/3.18 


25.0/3.61 


Science 


23.9/4.15 


25.4/3.10 


25.6/3.35 


25.0/3.95 


Technology 


24.2/3.55 


24.1/3.26 


24.5/3.90 


25.0/3.28 


Passage 3 




Educational Level 




Faculty 


Low 


Lowish 


Medium 


High 


Arts 


18.4/7.58 


21.4/5.91 


23.3/5.86 


25.2/4.8 


Social Sciences 


19.3/6.82 


19.7/7.08 


22.1/5.42 


23.0/5.27 


Maths 


19.1/6.52 


19.9/6.45 


21.4/6.21 


22.3/6.17 


Science 


18.2/7.25 


20.1 / 6.25^22.3/ 5.78 
19.3/6.3 y 19.8/7.6 


24.3/5.40 


Technology 


17.3/7.82 


21.3/5.85 


Vocabulary 




Educational Level 




Faculty 


Low 


Lowish 


Medium 


High 


Arts 


24^5 / 5.02 


25.2/4.92 


25.8/5.26 


27.6/4.71 


Social Sciences 


21.4/6.58 


23.3/5.89 


24.2 / 5.27 


25.1/6.42 


Maths 


22.6/7.17 


22.2/4.7 


22.7/5.34 


24.5/6.27 


Science 


22.3/6.25 


23.9/4.78 


24.0/4.78 


26.1 / 4.13 


Technology 


21.9/5.00 


23.0/4.15 


22.7/5.68 


24.0/4.51 




16 



IS 



Summary of ANOVA results comparing early with late returns 



Passage 1 

Source ofVariance DF 

Date Returned 9 

Passage 2 

Source of Variance DF 

Date Returned 9 

Passage 3 

Source of Variance DF 

Date Returned 9 

Vocabulary 

Source of Variance DF 

Date Returned 9 



F Value 


Significance Level 


2.15 


p<0.02 


F Value 


Significance Level 


2.62 


p<0.005 


F Value 


Significance Level 


7.53 


p<0.0001 


F Value 


Significance Level 


8.23 


p<0.0001 



DF = degrees of freedom 



Table 8 

Means for students by date of return 
(Tukey’s Studentized RangeTest) 

There were 10 Groups of c.200 Students; Group 1 returned the earliest. 
Group 10 returned the latest. Only significant differences are listed (p<0.05). 

Passage 1 

There were no significant differences. 

Passage 2 

Difference Between Means 



Group 2 v Group 5 


1.2 


Group 3 v Group 5 


1.19 


Passage 3 




Group 1 v Group 7 


2.24 


Group 1 v Group 8 


2.16 


Group 1 v Group 9 


3.45 


Group 1 v Group 1 0 


2.78 


Group 2 v Group 7 


2.09 


Group 2 v Group 8 


2.02 


Group 2 v Group 9 


3.31 


Group 2 v Group 10 


2.63 


Group 3 v Group 8 


2.75 


Group 3 v Group 9 


4.05 


Group 3 v Group 10 


3.37 


Group 4 v Group 9 


2.75 


Vocabulary 




Group 1 v Group 5 


2.20 


Group 1 v Group 6 


2.28 


Group 1 v Group 7 


2.80 


Group 1 v Group 8 


2.80 


Group 1 v Group 9 


3.64 


Group 1 v Group 10 


3.09 


Group 2 v Group 9 


2.30 


Group 2 v Group 10 


1.76 


Group 3 v Group 7 


2.05 


Group 3 v Group 8 


2.06 


Group 3 v Group 9 


2.89 


Group 3 v Group 10 


2.35 


Group 4 v Group 7 


1.88 


Group 4 v Group 8 


1.89 


Group 4 v Group 9 


2.72 


Group 4 v Group 10 


2.18 




18 



20 



Text & Readers Programme 

Publications 



Technical Reports 

1 : The revolution in print technology 
2: A postal survey of OU students’ reading skills 
3: Results of the survey of OU students’ reading skills 
4: OU students’ reading skills and final registration 

Newsflash 

1 : OU students’ reading skills 
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News from IET 



Text & Readers Programme 

OU Students’ Reading Skills 

results of a survey of 1995 entrants 

Michael Macdonald-Ross & Bernard Scott 



Birth of a new survey 

IET’s Michael Macdonald-Ross & Bernard Scott 
have sampled about 10%(3,000) of this year’s entering 
students from the five foundation courses. The survey 
used four tests of basic reading skills. The tests were 
three ‘cloze’ test' <traded in difficulty, and one 
vocabulary test. 

Cloze tests are a kind of completion test where every 
fifth or sixth word is deleted and the subject is asked 
to fill in the blanks. Research has shown this to be 
one of the most reliable ways of assessing reading 
skills. 

This is believed to be the first time reading skills have 
been measured by postal methods, and is one of the 
largest surveys of reading skills in the adult UK 
population. 

Of the 3,000 questionnaires sent out, just over 2,000 
were returned to IET, where they were scored and 
coded by IET’s Survey Research Office. 

The tests 

The first two tests were developed by ALBSU, the 
Adult Literacy and Basic Skills Unit, and used on 
our survey by their kind permission. The third test 
was developed by us from a THES book review. 

In cloze tests of this kind scores of 60% or above 
show the person can read the passage satisfactorily 
- that is, without assistance. Scores below 40% show 
inadequate comprehension and intermediate scores 
(between 40% & 60%) show partial comprehension. 
The vocabulary test was developed by the late Hunter 
Diack, and we are grateful to Mrs Diack for her 
permission to use her husband’s material. 

Passage 1 

The first passage was designed to identify readers 
who have trouble with basic functional literacy. 
As expected, almost all the OU students performed 
well on this test. 

Passage 2 

On passage 2, a more difficult passage, almost a third 
of students showed partial comprehension and just 
O yer two thirds showed satisfactory comprehension. 

eric 



First passage: 

Safe as houses 

w c Anfc of oar heme** ufc ptaoe to be md are matt wxucd wiea kmcok >oa 

oat of fee toaee Aanwtca Aey say ai boote. la tax pcopte Ac from 

aoode ms m btme ewr tear thm MC cm Ac road or * war*. 

ire Ac cmq of Acac • Ac fc*c? Maay pc«**e way 

offtea* Ae*r«A» bwi a Ua taorc people 

Croat CWkaf tka (not my caa*^Triex»caA»e,io*cCjnl 

amoA by c MAa md adate . or 

wdw^dowt.Poan— tcutoo UJamorAaA.Tb«a«y 

(iKmmcdxmaatrmbomcWmULmca u demx* 

■rnlt Food 4 ttoo » oo n w aayy. 

Sootto fc*«**tiec>A»ofca«w mwedt- 

cafe. People at u wort tow aad km *ort taw .Ticy 

Aeroforc *pe*4 wort m m tatcDtikMUio matmem 

«c Mater of m Acfcoac. The two vaAcntote powfaof 

p topic yoMf dOAta (eapeoatty p*c-«too< ) aad Ac 

ddety. Tfceae mtwtamcm Ac bom tor owa p ir . 

o*tocAkfcraw*o w tdmoL or wAta . acOMWtAwt 

*c . The dttorty are weaker aad itower rcaca m. Yaw* 




Passage 1: All Students 




Scores 



Passage 2: All Students 
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Passage 3 

We developed Passage 3 so that students could try 
their hand at some fairly tvpical academic prose. 
The results are striking. 

Almost a third of students showed inadequate 
comprehension, and nearly two thirds showed partial 
comprehension. In all, 95% of all who responded 
showed some degree of inadequacy in their 
comprehension of this passage. What is more, this 
passage is by no means the most difficult they will 
meet in their career at the OU. 

We predict that many of the students joining us in 
the foundation courses will have some difficulty 
with the prose in our foundation courses. 

We believe that students who scored poorly on our 
third passage will be much more likely to drop out 
during the year than those who scored over 60%. 
The progress of all students in the 3,000 sample will 
be followed through the year to test this prediction. 

Difficulty of the passages 

Opposite is a scale showing the relative difficulty of 
the three passages as measured by the Flesch Reading 
Ease formula, a good predictor of the difficulty of 
prose for adult readers. We will be measuring the 
readability of samples from the foundation course 
texts to see what level of difficulty they may pose to 
our students. 

Vocabulary test 

The results of the vocabulary test were consistent 
with the cloze tests. To be comfortable with prose as 
difficult as passage 3 students would need to score 
about 70% or more on the vocabulary test. 

Differences between faculties 

The results for all faculties follow a similar pattern. 
Arts students have the highest average scores, and 
Technology students the lowest. These differences 
are shght, but they are highly significant statistically . 
Our data also show a relationship between students’ 
previous highest educational level and their 
performance on these reading tests. 



Passage 3: All Students 




Flesch Reading Ease Scale 
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Vocabulary: All Students 



200 - 70% 




Scores 

Passage 3 by Faculty 



The future 

We see our work as a way to improve the quality of 
O IJ courses, and a way to improve student retention. 
W« hope to follow our sample through their OU 
careers. Do their reading skills unprove, and if so by 
how much 0 

It would be interesting to have data on the readability 
of preparatory courses, and on students who enroll 
for these courses. Surveys of Business School 
students (especially European students for whom 
English is a second language) and Health & Social 
O Welfare students are two more ideas for the future. 
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